Support multiple prompts - #468
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for loading generation prompts from a file (prompt_file) across multiple video generation scripts (generate_ltx2.py, generate_ltx_video.py, and generate_wan.py), alongside updating configuration files and improving GCS upload paths. It also fixes latent decoding in wan_pipeline.py by concatenating addressable shards. The review feedback highlights a critical performance bottleneck and a NameError in generate_ltx_video.py caused by initializing the pipeline inside the prompt loop. Additionally, the reviewer recommends raising a ValueError in max_utils.py to prevent downstream IndexError crashes when no prompts are found, and suggests removing an unused last_out variable in generate_ltx2.py.
| for prompt_idx, current_prompt in enumerate(prompts): | ||
| prompt_word_count = len(current_prompt.split()) | ||
| enhance_prompt = prompt_enhancement_words_threshold > 0 and prompt_word_count < prompt_enhancement_words_threshold | ||
|
|
||
| pipeline = LTXVideoPipeline.from_pretrained(config, enhance_prompt=enhance_prompt) | ||
| if config.pipeline_type == "multi-scale": | ||
| pipeline = LTXMultiScalePipeline(pipeline) | ||
| conditioning_media_paths = config.conditioning_media_paths if isinstance(config.conditioning_media_paths, List) else None | ||
| conditioning_start_frames = config.conditioning_start_frames | ||
| conditioning_strengths = None | ||
| if conditioning_media_paths: | ||
| if not conditioning_strengths: | ||
| conditioning_strengths = [1.0] * len(conditioning_media_paths) | ||
| conditioning_items = ( | ||
| prepare_conditioning( | ||
| conditioning_media_paths=conditioning_media_paths, | ||
| conditioning_strengths=conditioning_strengths, | ||
| conditioning_start_frames=conditioning_start_frames, | ||
| height=config.height, | ||
| width=config.width, | ||
| padding=padding, | ||
| ) | ||
| if conditioning_media_paths | ||
| else None | ||
| ) |
There was a problem hiding this comment.
The pipeline initialization (LTXVideoPipeline.from_pretrained) and conditioning preparation are currently placed inside the prompt loop. This means the entire 13B model and conditioning inputs will be reloaded and recompiled for every single prompt, causing a massive performance bottleneck and potential out-of-memory errors.
Additionally, prompt_enhancement_words_threshold is referenced on line 189 but its definition was removed, which will cause a NameError at runtime.
Moving the pipeline initialization and conditioning preparation outside the loop resolves both issues. We can determine if prompt enhancement is needed for any of the prompts beforehand, and then pass the individual enhance_prompt flag to the pipeline call inside the loop.
prompt_enhancement_words_threshold = getattr(config, "prompt_enhancement_words_threshold", 0)
any_enhance_prompt = any(
prompt_enhancement_words_threshold > 0 and len(p.split()) < prompt_enhancement_words_threshold
for p in prompts
)
pipeline = LTXVideoPipeline.from_pretrained(config, enhance_prompt=any_enhance_prompt)
if config.pipeline_type == "multi-scale":
pipeline = LTXMultiScalePipeline(pipeline)
conditioning_media_paths = config.conditioning_media_paths if isinstance(config.conditioning_media_paths, List) else None
conditioning_start_frames = config.conditioning_start_frames
conditioning_strengths = None
if conditioning_media_paths:
if not conditioning_strengths:
conditioning_strengths = [1.0] * len(conditioning_media_paths)
conditioning_items = (
prepare_conditioning(
conditioning_media_paths=conditioning_media_paths,
conditioning_strengths=conditioning_strengths,
conditioning_start_frames=conditioning_start_frames,
height=config.height,
width=config.width,
padding=padding,
)
if conditioning_media_paths
else None
)
for prompt_idx, current_prompt in enumerate(prompts):
prompt_word_count = len(current_prompt.split())
enhance_prompt = prompt_enhancement_words_threshold > 0 and prompt_word_count < prompt_enhancement_words_threshold| if not prompt_file_path: | ||
| if default_prompt: | ||
| return [default_prompt] | ||
| return [] | ||
|
|
||
| prompt_file_path = prompt_file_path.strip() | ||
| if not prompt_file_path: | ||
| if default_prompt: | ||
| return [default_prompt] | ||
| return [] |
There was a problem hiding this comment.
If both prompt_file_path and default_prompt are empty or falsy, load_prompts currently returns an empty list []. This will cause downstream scripts (like generate_ltx2.py, generate_wan.py, and generate_ltx_video.py) to crash with an IndexError when they attempt to access prompts[0] for warmup or generation.
Raising a ValueError here prevents these silent failures and provides a clear error message to the user.
| if not prompt_file_path: | |
| if default_prompt: | |
| return [default_prompt] | |
| return [] | |
| prompt_file_path = prompt_file_path.strip() | |
| if not prompt_file_path: | |
| if default_prompt: | |
| return [default_prompt] | |
| return [] | |
| if not prompt_file_path: | |
| if default_prompt: | |
| return [default_prompt] | |
| raise ValueError("No prompts found. Both prompt_file_path and default_prompt are empty.") | |
| prompt_file_path = prompt_file_path.strip() | |
| if not prompt_file_path: | |
| if default_prompt: | |
| return [default_prompt] | |
| raise ValueError("No prompts found. Both prompt_file_path and default_prompt are empty.") |
| last_out = None | ||
| if not is_multi_prompt: | ||
| prompt = [prompts[0]] * batch_size | ||
| negative_prompt = [negative_prompt_str] * batch_size | ||
| out = call_pipeline(config, pipeline, prompt, negative_prompt) | ||
| last_out = out |
There was a problem hiding this comment.
The variable last_out is initialized and assigned but never used. The timing summary on line 478 still references out. You can safely remove last_out to keep the code clean. Please also remove the assignment last_out = out on line 438.
| last_out = None | |
| if not is_multi_prompt: | |
| prompt = [prompts[0]] * batch_size | |
| negative_prompt = [negative_prompt_str] * batch_size | |
| out = call_pipeline(config, pipeline, prompt, negative_prompt) | |
| last_out = out | |
| if not is_multi_prompt: | |
| prompt = [prompts[0]] * batch_size | |
| negative_prompt = [negative_prompt_str] * batch_size | |
| out = call_pipeline(config, pipeline, prompt, negative_prompt) |
1 participant
This PR adds supports for enabling inference scripts to provide multiple prompt-inputs to generate videos via txt files or GCS locations.
It will be useful for integrating VBench evals into this repo.